Color signal processing

ABSTRACT

A method of adjusting the N components of an N component color signal (such as the 3 components of an RGB color signal) with respect to color temperature of illuminating light when imaging a colored subject, comprising the step of computing the difference between each of the components of the color signal for different temperatures of illuminating light, using an algorithm based on the spectral sensitivity of the imaging means and Planck&#39;s equation defining black body illumination to obtain N components whose values are independent of the color temperature of the illuminating light. Normalisation of the adjusted values renders the values independent of intensity of illumination and can also reduce the processing time for further processing of the signals. A technique for compensating for gamma correction which is commonly built in cameras, is disclosed.

FIELD OF INVENTION

This invention concerns the processing of colour signals such as RGBsignals from a camera or scanner, so as to remove dependency onilluminant colour temperature.

INTRODUCTION

The light reaching the eye is a function of surface reflectance andilluminant colour. Yet, the colours that are perceived depend almostexclusively on surface reflectance; the dependency due to illumrinantcolour is removed through colour constancy computation. As an example,the white page of a book looks white whether viewed under blue sky orunder artificial light. However, the processes through which colourconstancy is attained are not well understood. Indeed, the performanceof colour constancy algorithms in computer vision remain quite limited.

The colour constancy problem can be solved if red, green and blue sensorresponses, or RGBs, for surfaces seen under an unknown illuminant can bemapped to corresponding RGBs under a known reference light. Despitesignificant effort, this general 3-dimensional colour constancy problemhas yet to be solved. It has been argued that the 3dimensional problemis in fact too difficult to solve: there is an intrinsic ambiguitybetween the brightness of an illuminant and the lightness of a surfaceand so dark surfaces viewed under bright lights reflect the samespectral power distribution as highly reflective surfaces under dimmerlight. This argument is taken on board in almost all modern colourconstancy algorithms; modern algorithms attempt only to recoverreference chromaticities.

The chromaticity constancy problem has proven to be much more tractable.In Colour in perspective (IEEE Transactions, pages 1034 to 1038, October1996), Finlayson made two important observations. First, was that thegamut of possible image chromaticities depended on the illuminant colour(this result follows from previous work on 3-dimensional RGB gamuts) andsecond, that the illuminant colour was itself quite limited. Thechromaticities of real illuminants tend to be tightly clustered aroundthe Planckian locus. In Finlayson's algorithm an image chromaticity issaid to be consistent with a particular light if it is within the gamutof all possible chromaticities observable under that light. Usually asingle chromaticity will be consistent with many lights; but differentchromaticities are consistent with different sets of lights.Intersecting all the illuminant sets results in an overall set offeasible illuminants: illuminants that are consistent with all imagechromaticities together and at the same time. Typically, the set offeasible illuminants is quite small and selecting the mean or medianilluminant from the feasible set leads to good colour constancy.Unfortunately, when colour diversity is small, the feasible set can belarge. In this case it is quite possible that an incorrect illuminantwill be selected and when this happens poor colour constancy results.

In more recent work, the ill-posed nature of the colour constancyproblem has been tackled using the tool of Bayesian probability theory.Given knowledge of typical scenes it is possible to calculate theprobability of observing a particular chromaticity under a particularlight. This prior information can then be used to calculate thelikelihood of lights given the chromaticities in an image. While thisapproach delivers much better colour constancy the problem of low colourdiversity, though certainly diminished, still remains. For scenescontaining small numbers of surfaces (1, 2, 3 or 4) many illuminants canbe equally likely.

Whether or not the problem of such low colour diversity is importantdepends on applications. In digital photography, typical pictures are ofcolour rich scenes and so the probabilistic approach works well.However, in computer vision interest lies in analysing colour in colourdeficient scenes. Beginning with Swain and Ballard (Colour Indexing,International Journal of Computer Vision, 7(11) 11–32, 1991), manyauthors have attempted to use the distribution of colours orchromaticities in an image as a cue to image content in general andobject recognition in particular. This idea works well when lightingcolour is held fixed but can fail spectacularly when illumination isallowed to vary. Swain conjectured that colour constancy preprocessingwould solve the varying illumination problem. Unfortunately, because theobjects to be recognised sometimes have low colour diversity—manybranded products, such as Campbell's soup (used in Swain's originalexperiments), have just 1 or 2 colours—the colour constancy problem isnot easy to solve. Indeed, Funt et al (Is machine colour constancy goodenough? Fifth European Conference on Computer Vision (Vol. II), pages445–459) tested a variety of chromaticity constancy algorithms andconlcuded that none of them rendered chromaticity a stable enough cuefor recognition.

The failure of the colour constancy preprocessing in object recognitionhas inspired the colour invariant approach. Colour invariants aregenerally functions of several image colours designed so that termsdependent on illumination cancel. As an example, if (r₁, g₁, b₁) and(r₂, g₂, b₂) denote camera responses corresponding to two scene pointsviewed under one colour of light then (αr₁,βg₁, γb₁) and (αr₂, βg₂, γb₂)denote the responses induced by the same points viewed under a differentcolour of light (assuming the camera sensors are sufficientlymarrow-band). Clearly, it is easy to derive algebraic expressions whereα, β and γ (and so illumination) cancel:

$\left( {\frac{r1}{r2},\frac{g1}{g2},\frac{b1}{b2}} \right).$

Indexing on colour-ratios (and other invariants), have been shown todeliver illuminant independent object recognition. Yet, this approachsuffers from three intrinsic problems. First, because spatial context isused, invariant computation is sensitive to occlusion. Second,invariants can only be calculated assuming there are two or more coloursadjacent to one another (not always true). Third, invariants can becalculated post-colour constancy computation but the converse is nottrue: colour constancy adds more information if it can be computed. Tounderstand this last case, consider an image taken under an unknownillumination. A perfect colour constancy algorithm can, by definition,map image colours to corresponding colours under the reference light: itis possible then to calculate the full 3-dimensional absolute colour ateach point in an image. In contrast, colour invariants cancel outvariation due to illumination by exploiting spatial context e.g. underfairly reasonable conditions the ratio of adjacent RGBs is illuminantindependent, The result is that absolute colour information isconfounded: the colour of a pixel can only be known relative to theneighbourhood of the calculated invariant. Clearly, one might calculaterelative information given absolute values (i.e. given the output of acolour constancy algorithm) but the converse is not true. It is in thissense that colour constancy computation delivers more information.

BACKGROUND OF THE INVENTION

The foregoing can be summarised as follows:

The image formed by light reflected from an object inter alia dependsupon two factors: (a) the intensity of illumination incident upon theobject (and hence the intensity of reflected light), and (b) thewavelength(s) of illumination incident upon the object, (more usuallyidentified as the “colour temperature” (T) of the illumination). Factor(b) determines the wavelengths of the reflected light, and therefore thecolours in the image formed, while factor (a) determines the brightnessof the image.

Clearly therefore, the same object may form different images when viewedby cameras or other image-forming devices, under different intensities,and/or wavelengths of illumination. In the case of daylight thesefactors (a) and (b) will tend to alter, for example, with the time ofday, or season. If artificial light is employed, even greater variationscan arise. This represents a problem for any object and/or imageanalysing and/or recognition system which relies on processing coloursignals for analysis from a camera such as the RGB signals from a threesensor camera.

It is possible to allow for variations in intensity of illumination, andvarious techniques are known and described in the art. As is well knowncolour can be described mathematically by a three component vector R, G,B. The output of a three sensor camera viewing an object provides thesethree components, the magnitude of each of which is indicative of thequantity of light reflected by the object at the red, green and bluewavelengths.

Although the nomenclature suggests single wavelength resolution, in factmost camera sensors are responsive to a broad band of wavelengths ineach of the red, green and blue regions of the visible spectrum, andsuch sensors are referred to as broad band sensors. However, camerashave been developed with a very limited response band width towavelengths in the red, green and blue regions of the spectrum, byutilising in such cameras narrow band sensors, i.e. sensors that onlyrespond to a narrow band of wavelengths in each of three regions of thespectrum.

Under different intensities of light the RGB signal varies in magnitude.Thus if the intensity scales by a factor k, then the camera signalequals (kR, kG, kB). By normalising the RGB vector in relation tointensity, the dependency due to intensity (k), can be removed.

Various techniques exist in the literature for removing k. For instanceone may divide the RGB vector by (R+G+B). This results in chromaticitynormalisation. Equally, one may divide by the square root of the sum ofthe squares of the RGB signal i.e. (R²+G²+B²). If R+G+B=1, thenB=1−(R+G) (i.e. given R and G it is possible to calculate B, so thethird parameter can be written in terms of the other two parameters). Ingeneral, any magnitude normalised RGB vector can be encoded by twoparameters.

Whilst various such methods are able to remove variations in an imagearising from fluctuations in intensity of illumination, it has notgenerally been possible to correct for variations in colour temperatureof the illumination.

One object of the present invention is to provide a method to providefor such colour temperature correction.

It is a further object of the present invention to bridge the gapbetween the classical colour constancy computation and the invariantapproach referred to in the Introduction section.

The invention has an additional object to provide a method of producingfrom colour corrected (and preferably also intensity corrected) colourimage signals, signals which will produce a grey scale image which issubstantially the same in terms of grey scale and brightnessirrespective of the illumination of the original object producing theimage.

SUMMARY OF THE INVENTION

The invention lies in a method of adjusting the N components of an Ncomponent colour signal (such as the 3 components of an RGB coloursignal) so as to produce component values which are independent of thecolour temperature of the illuminating light comprising the step ofcomputing for each component a new value using the original componentvalue and one other component value and an algorithm incorporating amathematical operator whose value is related to the spectral sensitivityof the imaging means to different colour temperature illuminants.

The invention also lies in a method of processing component signalsmaking up an N component colour signal (such as RGB signals of a 3component colour signal) defining an image of an illuminated colouredsubject to produce modified signal components defining an image of thesame subject as if illuminated under light of a different colourtemperature, comprising the steps of:

-   1) generating the N-(RGB etc.) components of a colour signal    corresponding to an illumination of the subject using illumination    of colour temperature T₁;-   2) forming the N log values (R′ G′ B′ etc.) of the N-RGB etc.    components of the colour signal so obtained; and-   3) computing the N log values (R*, G*, B* etc.) of the N-RGB etc.    components of the colour signal for a different illumination colour    temperature T₂ using the equation:    (R*G*B* . . . )=(R′G′B′. . . )+X(u, v, w, . . . )    where X is proportional to (T1–T2) and u, v, w etc. are constants    that depend only on the spectral sensitivities of the signal source    and Planck's equation defining black body illumination; and-   4) processing and/or storing and/or controlling a display device    using the N-(R*, G*, B*, . . . ) values or signals derived    therefrom, or converting the N log values (R*, G*, B*, . . . ) to    appropriate N-RGB etc. values for processing and/or display and/or    storing.

Considering the simple 3-component RGB case, and taking X into thebrackets reduces to:(R*,G*,B*)=(R′,G′,B′)+(Xu,Xv,Xw)

Further simplification:(R*,G*,B*)=(R′+Xu,G′+Xv,B′+Xw)

The above equation is really a way of writing three separate equations:R*=R′+Xu  (a)G*=G′+Xv  (b)B*=B′+Xw  (c)

Since X is constant for all three (a) to (c), Xv in (b) can be convertedto Xu by dividing (b) by v and multiplying by u throughout. The Xuexpression (which is the only temperature dependent element in (b) and(a) can be eliminated by subtracting

$\frac{u}{v}(b)$from both sides of

-   (a) giving on one side

$\left( {R^{*} - {\frac{u}{v}G^{*}}} \right)$and on the other side

$\left( {R^{\prime} + {Xu}} \right) - {\frac{u}{v} \cdot \left( {G^{\prime} + {Xv}} \right)}$(R′+Xu)−u(G′+Xv)/v.

The right hand side can be re-written as:(R′+Xu)−uG′/v−uXv/vand this reduces to:R′+Xu−uG′/v−Xuwhich equals (R′−uG′/v)

${{{ie}\mspace{14mu} R^{*}} - {\frac{u}{v}G^{*}}} = {{R^{}}^{\prime} + {\frac{u}{v}{G^{\prime}.}}}$

In a similar way

$B^{*} - {\frac{w}{v}G^{*}}$can be shown to equal

$B^{\prime} - {\frac{w}{v}{G^{\prime}.}}$

In the same way, by taking G*/v inside the brackets; we can re-write(R*, G*, B*)−G*/v(u,v,w), as(R*,G*,B*)−(uG*/v,vG*/v,wG*/v)

Simplifying again:

$\left( {R^{*} - {\frac{u}{v}G^{*}}} \right),\left( {G^{*} - {\frac{v}{v}G^{*}}} \right),\left( {B^{*} - {\frac{w}{v}G^{*}}} \right)$and this equals:

$\left( {R^{*} - {\frac{u}{v}G^{*}}} \right),O,\left( {B^{*} - {\frac{w}{v}G^{*}}} \right)$

Previously, it has been shown that

$R^{*} - {\frac{u}{v}G^{*}\mspace{11mu}{and}\mspace{14mu} B^{*}} - {\frac{w}{v}G^{*}}$are independent of X and hence temperature. So, (R*, G*, B*)−G*/v(u, v,w)=(x, O, y) (where x and y denote illuminant independent quantities),or (R*, G*, B*)−R*/u(u, v, w)=(O, x, y), by analogy to the arguments setforth above.

Values for the parameters u, v and w, of, for example, a colour cameraor scanner may be obtained using the following method. Under a firstilluminant (say tungsten), take a picture of a uniform colour patch.Denote the log of the RGB components as (R′₁, G′₁, B′₁). Now change theilluminant (say to daylight) and obtain a second RGB signal and convertto log values (R′₂, G′₂, B′₂) again for the RGB due to the secondilluminant.

From the foregoing and relying on Planck's theory, the log RGB valuesunder lights 1 and 2 are related as follows:(R′ ₁ ,G′ ₁ ,B′ ₁)=(R′ ₂ ,G′ ₂ ,B′ ₂)+X(u,v,w)

Without loss of generality, we can set X=1. We can do this because X issimply a scalar value which disappears (is factored out) if the RGBsignal values are expressed in a manner which is independent of colourtemperature. Under this assumption:u=R′ ₁ −R′ ₂ ,v=G′ ₁ −G′ ₂ ,w=B′ ₁ −B′ ₂

In practice a preferred extension is to select values for u, v, and wbased on a number of different observations of the same surface underdifferent lights and of other coloured surfaces under similar differentilluminants (lights). This can be done statistically and optimised, aswill be described later.

The variable T presented in the text refers to colour temperature. Fortypical illuminants, T will be in the range 2000K (orangish light) to10000K (bluish light). 5000K correlates with whitish light.

The colour signal to be processed may be obtained from a camera, amultispectral imaging device, or a scanner, or may be a computergenerated RGB etc. signal.

The method is typically applied to the colour component values of eachpixel making up a multi-pixel image.

In the case of a 3 component RGB signal, the RGB vectors can bedescribed in terms of their orientation with respect to three orthogonalR, G and B axes. For any vector, its orientation will define theresulting colour produced by the signals. Two orientations (or angles)suffice to describe a three-dimensional vector (irrespective of itsmagnitude).

When applied to an image containing a large number of pixels the timerequired for adjustment and processing can be reduced and the signalrendered intensity independent by normalising the RGB etc. values withrespect to one of the values—(e.g. with respect to the G value), in thatfor example (R/G, 1, B/G) can be written as (R/G, B/G), in the case of a3 component RGB signal.

In accordance with the invention the mathematical operator employed toreduce the number of components from say 3 to 2, is independent of thecolour temperature T of the illumination.

The invention allows the log value R*, G*, B* of each of the componentsof a 3-component RGB signal to be represented by three log values:(R*−uG*/v, (B*−wg*/v) and a third term which will always be zero; thetwo non-zero values being independent of the illumination colourtemperature.

The invention also allows these three log values to be expressed as athree-value vector (R*, G*, B*)−G*/v (u, v, w) which can be written inthe form (x, 0, y), where again the x and y components are independentof the colour temperature of light illuminating (or deemed to beilluminating) a subject from which the colour signal is obtained.

In each case the co-ordinates or vector components are two numbers thatare independent of the temperature of the illuminant, and in each casenormalising is achieved in accordance with the invention by usingalgebra to find expressions which cancel out T and do not rely on thevalue of X.

The invention also provides a method of processing the N componentsignals making up a colour signal (such as RGB signals of a 3 componentcolour signal) defining an image of an illuminated coloured subject toproduce modified signal components defining an image of the same subjectas if illuminated under light of a different colour temperture (T),comprising the steps of:

-   (1) normalising the N components of the colour signal with respect    to one of the components (e.g. G) to provide (N−1) co-ordinates    (e.g. R/G, B/G in a 3 component RGB signal) which are independent of    the illumination intensity,-   (2) computing the log values of these (N−1) co-ordinates, (such that    log(R/G)=R′ and log (B/G)=B′ in the case of an RGB signal),-   (3) adjusting the values of (R′, B′, etc.) to produce new values    (R*, B*, etc. ) corresponding to the values which would apply if the    colour temperature of the illumination is T₂ using the equation (R*,    B*, etc.)=(R′, B′, etc.)+X(a, b, etc.) where X is proportional to    (T₁–T₂) and a and b etc. are constants that depend only on the    spectral sensitivities of the source and Planck's equation defining    black-body illumination; and-   (4) converting the values (R*, B*, etc.) to new (RGB etc. ) values    for processing and/or display.

In accordance with the invention a reduction in the number of values tobe processed for rendering purposes is achieved by mathematicallynormalising the expression (R′, B′, etc.)+X(a, b, etc.) so as to beindependent of the variable T.

In the case of a 3 component RGB signal, one such mathematicalnormalisation results in the expression being reduced to the co-ordinate(R*−aB*/b), which is independent of colour temperature.

As a result of an alternative normalisation, in the case of an RGBsignal, the expression can be reduced to the vector (R*, B*)−B*/b(a, b)which is also illuminant invariant.

In this latter case the co-ordinates are in the form (x, 0), indicatingthat with appropriate normalisation, a colour signal can be reduced to asingle number x that is independent of the colour temperature T of theillumination.

Any algebraic operation of (R′, B′) that removes dependency on T is acolour temperature normalisation, and is within the scope of thisinvention.

It is a feature of the invention that all temperature normalisedco-ordinates (derived by any algebraic argument) can be transformed intoone another.

If the source has Delta function sensitivities and can therefore bethought of as having narrow-band sensors, then (a, b) can be foundanalytically (they follow from Planck″ equation). For wide-band sensorsources (a, b) can be found experimentally.

As a precursor to any method involving the processing methods describedabove, it is of course necessary to calibrate the source in advance anddetermine the (a, b) values for the source.

The invention thus lies in a method of processing a colour signal from acolour signal source so as to eliminate dependency on colour temperatureand/or intensity variation by using a mathematical algorithm derivedfrom the source sensitivities to different wavelengths of light, in asignal processing computer supplied with the colour signal, to providenew (corrected) values for the component making up the colour signalwhich are independent of the colour temperature of the lightilluminating (or deemed to illuminate) the subject described by thecolour signals.

The invention also lies in a computer when programmed so as to processRGB signals from a colour signal source so as to produce illuminationcolour temperature and/or intensity corrected signals from the three RGBsignals, to form an illumination colour and/or intensity corrected RGBsignal for image processing and/or for display and/or shape.

The invention also lies in the step of converting corrected signals asaforesaid into a grey scale signal for processing and/or storage and/orfor forming a grey scale image in place of a colour image in a display.

The invention enables a 1-dimensional colour co-ordinate to be expressedas a function of the RGB or chromaticity, so as to remove dependence onthe illumination colour temperature by converting a 2 component coloursignal into a 1 component grey-scale signal, and a comparison isperformed by the step of mapping, the grey-scale values to those of asimilar subject observed under reference lighting conditions.

The invention has revealed that not only does there exist a colourco-ordinate where colour constancy computation is easy, there exists aco-ordinate where no computation actually needs to be done.

The grey-scale signal can factor out all dependencies due toillumination intensity and illumination colour temperature, if thechromaticities of lights lie on the Planckian locus and the camerasensors sample light like Dirac Delta functions (they have narrow-bandsensitivities). If so illumination change in log-chromaticity space istranslational and it follows that log-chromaticities, which areintensity independent, also translate under a change in illuminationcolour. Thus (In R/G, In B/G) becomes (In R/G, In B/G)+(a, b) under adifferent illumination.

It is important to note that the translational term (a, b) must be thesame for all RGBs.

It is also a requirement that the translational term for differentilluminations can always be written as(αa, αb)where a and b are fixed constants and a depends on illumination. Thatis, illumination change translates log-chromaticities in the samedirection.

It follows that the co-ordinate axis orthogonal to the direction ofillumination variation, y=−(a/b)x, records only illuriinant invariantinformation and there exist constants r₁ and r₂ such that theco-ordinate r₁1n R/G+r₂ln B/G is independent of illumination. Of coursereal illuminations will rarely lie exactly on the Planckian locus norwill camera sensitivities be exactly narrow-band. However, experimentsdemonstrate that the invariant computation is robust to departures fromeither of these prerequisites.

In the foregoing no reference is made to solving for colour constancy.Rather the invariant colour co-ordinates calculated under allilluminations, including the reference light, do not change. That is,invariant computation at a pixel, and 1-dimensional colour constancy,are two sides of the same coin.

Using a derived invariant co-ordinate provided by the invention, allowscolour based object recognition to be performed.

To this end objects are represented by the distributions of theinvariant co-ordinates calculated from colour images of the objects; ineffect using grey-scale histograms (since the invariant co-ordinate canbe coded as grey-scale). Recognition is performed by distributioncomparison; that is an unknown distribution is compared with objectdistributions stored in a database and the closest match identifies theunknown.

In one experiment a database was used of 11 images of objects viewedunder a single illuminant from the Simon Fraser calibrated imagedataset. Unknown images were of the same object, viewed under 4different coloured illuminations. It was found that comparison andindexing on invariant co-ordinate distribution delivered near perfectrecognition. By comparison indexing on chromaticity distributions(calculated post 2-dimensional colour constancy processing) deliveredmarkedly poorer performance.

The invention also lies in a method of image analysis comprising thesteps of comparing illumination colour temperature and/or intensitycorrected signals as aforesaid or summations or other computationalcombinations of some or all of the corrected signals, with stored valuesof such signals and/or similar summations or other computationalcombinations thereof, to permit object recognition.

The invention also lies in a computer programmed to perform the saidcomparison.

In a method involving the generation of a grey scale image, thecomponents making up the grey scale image are preferably stored as agrey scale signal for comparison purposes in object identification orcomparison.

The intensity of light generated by a physical device as an output inresponse to an applied input signal (such as an R, G, B signal) is oftennot a linear function of the input signal. This is particularly relevantfor cathode ray tubes (CRT) (and may also apply to solid state displaypanels), as are used in televisions and computer monitors. Such devicesuse an R, G, B signal to control an electron beam in a CRT or the colourto be produced by an addressed pixel of a display panel. Typically thisnon-linearity is modelled as a power function: that is, if the appliedsignal in, for example, the red channel is d_(r), then the resultingintensity of light R generated by the display is given by:R=(d_(r))^(γ). If the value of γ is the same for the red, green, andblue channels of the device then a single number characterises thedevice. In general the value of γ may be different for the threechannels, in which case, three different factors α, β, and ρ arerequired to characterise the device.

Given the inherent non-linearity of CRTs and other display devices, ifan image is displayed on a monitor, this non-linearity must be accountedfor so that a displayed image matches the original scene as closely aspossible. This can be done by correcting the output signal from thecamera in the following way. If the camera records a signal R_(c), andif this value is linearly related to light intensity, then applying aninverse power function to this signal: R¹ _(c)=(R_(c))^((1/γ)), ensuresthat if this signal (R¹ _(c)) is used as the input to the CRT, theresulting light intensity will be linearly related to the original lightintensity of the scene. Accordingly, it is common practice for cameramanufacturers to apply a power function γ, to the signals in each of thethree channels, to compensate for the expected non-linearity of adisplay device such as a CRT based display or monitor.

Generally, it is desirable to maintain a linear relationship between theR, G, B signal and the resulting image, especially in relation tocomputer-enhanced or other computer processed images. Accordingly, if itis desires to perform a data processing step before, or instead of,displaying the image on a monitor or other image-forming device, it isdesirable to compensate for a deliberately-applied γ factor to theoutput signal from a camera (or indeed any γ factor which may be aninherent characteristic of the camera), in order to maintain a linearrelationship.

Invariance to γ can be obtained by dividing the log camera responses bythe average of the log colour responses in a local neighbourhood of theimage (e.g. in all or some of the immediately adjacent pixels), prior tocalculating the invariant co-ordinates. Alternatively the invariantco-ordinate can be calculated, as before, and then normalised by theaverage invariant co-ordinate in a local neighbourhood (e.g. for all orsome of the immediately adjacent pixels) of the image.

The invention therefore provides a method of normalising R^(α), G^(β),B^(ρ) signals from a camera in which a different γ factor applies toeach of the three channels, to remove dependency on the values of α, βand ρ. It is particularly preferred that the α, β and ρ normalisationmethod is employed in combination with the colour and/or intensity ofillumination methods defined above.

The invention is applicable to enhancement methods and apparatus sincethe log of the colour response is composed of three parts, a vector dueto the intensity of the illumination, a vector due to the illuminationcolour temperature, and a vector due to reflective properties of thesurface illuminated for the purpose of producing the image.

By changing the magnitude of the intensity vector, or the magnitude ofthe illumination colour temperature vector for each pixel in an image,one can change the effective illumination for each pixel in the finalimage.

Variation of the magnitude of the illumination colour temperature vectorproduces a variation in depth of colour but does not alter the nature ofthe colour. Thus the colour warmness or colour depth can be increased ordecreased by altering the magnitude of the illumination colourtemperature vector, and recreating the pixel using the altered vectorvalue. Preferably this is done by scaling all illumination colourtemperature vectors for all pixels in the image.

Enhancement is thus simplified by simply altering or scaling theillumination colour temperature vector to give warmer or colder colour.

In a similar way the illumination intensity vector can be increased ordecreased so as to brighten or darken the overall picture. This allowsenhancement of a final image without altering the illumination or theoriginal object producing the image.

The vector alteration may be performed on a frame basis, or on selectedregions of a frame made up of many pixels, or on a pixel by pixel basis.

The invention is not limited to a three-channel colour signal system butcan be applied to a multi-channel system such as a four-channel systemin which the R, G and B signals are supplemented by a Z signal. Thisrequires four sensors, the fourth sensor being sensitive to wavelengthscorresponding to the Z band. This may for example be another colour inthe visible range, or in the ultra-violet, or in the infra-red.

Illumination intensity dependence can be reduced by selecting a valuefor K such that (R+G+B+Z)/K=1. By dividing each of the vector values bythe value K, any one of the scaled vector values can be expressed interms of the other three scaled vector values, and the four coloursignals RGB and Z can be represented by three numbers by dividing eachof the R, G, B and Z signals by the sum of the RGB and Z signals andsolving for one of the RGB or Z values. Where the vector can be thoughtof as having four component vectors each having different anglesrelative to four colour axes, the four angle vectors can be reduced thethree angle vectors. Intensity can be normalised in a similar way bydividing each of the vector values R, B, G and Z by, for example, the Gchannel signal so as to produce R/G, B/G and Z/G and computing the logof each of the three quotients to give R′, B′ Z′. The expression (R′,B′, Z′)+1/T(a, b, c) can be algebraically or otherwise computationallynormalised so as to remove any dependency on the value of T.

Examples of three channel vectors have been given and it will be seenthat using the four channel values R, G, B and Z similar co-ordinatesand vectors can be defined which do not involve T, and by using a logchromaticity value, they will also not depend on illumination intensity.The same technique can readily be extrapolated to situations involvingany number (n) of colour channels (e.g. to a camera having n differentwavelength/waveband sensors).

The invention lends itself to various applications of which thefollowing are merely examples.

-   a. Checking the colour of foodstuffs or other processed materials at    the end of a manufacturing or processing line, and which may be    subjected to a different illumination for the checking process from    that in which they will be viewed by a consumer;-   b. Checking allegedly similar objects which nay be subjected to    different illuminations and identifying those which have a high    degree of similarity to a standard object;-   c. Distinguishing an object having a boundary relative to a    background more reliably, particularly under varying and/or    difficult light conditions such as when the boundary is shaded    and/or the illumination varies across the scene containing the    object which is being imaged;-   d. Creation of image databases by illuminating and imaging objects    with varying illumination and storing data relating thereto for    subsequent comparison purposes, in which the invention allows the    colour signals relating to the stored images to be normalised, as if    all the objects had been illuminated using the same wavelength of    illuminating light, and if required, also the same intensity;-   e. Looking for a face in pictures, images, or scenes, and tracking    same, such as in a communication system involving speech and    transmission of an image of the face of the person speaking, in    which the illumination of the face can vary.

MORE DETAILED DESCRIPTION OF THE INVENTION

The invention will now be further described with reference to theaccompanying drawings.

Figure Captions

FIG. 1: Normalised 2500K black-body radiator: exact equation (solidline), approximation (dashed line). Note in this case the two lines areon top of each other.

FIG. 2: Normalised 5500K black-body radiator: exact equation (solidline), approximation (dashed line).

FIG. 3: Normalised 10000K black-body radiator: exact equation (solidline), approximation (dashed line).

FIG. 4: Normalised 5500K CIE D55 standard daylight (solid line).Planckian illuminant (dashed line).

FIG. 5: CIE xy chromaticity diagram. Solid curved line is thechromaticity locus for all typical Planckian black-body lights. Fromleft to right illuminants begin bluish, become whitish then yellowishending in reddish light. Crosses (‘+’) denote chromaticities of typicalnatural and man-made lights.

FIG. 6: Perfect Dirac-Delta camera data (sensitivities anchored at 450nm, 540 nm and 610 nm). Log chromaticity differences (LCDs) for 7surfaces (green, yellow, white, blue, purple, orange and red) under 10Planckian lights (with increasing temperature from 2800K to 10000K).Variation due to illumination is along a single direction.

FIG. 7: Perfect Dirac-Delta camera data (sensitivities anchored at 450nm, 540 nm and 610 nm). Log chromaticity differences (LCDs) for 7surfaces (green, yellow, white, blue, purple, orange and red) under 10Planckian lights (with increasing temperature from 2800K to 10000K).LCDs (from FIG. 6) have been rotated so that x co-ordinate depends onlyon surface reflectance; the y co-ordinate depends strongly onillumination.

FIG. 8: CIE xy chromaticity diagram. Solid curved line is thechromaticity locus for all typical Planckian black-body lights. Fromleft to right illuminants begin bluish, become whitish then yellowishending in reddish light. Dots (‘.’) denote chromaticities of weightedaverages of Planckian locus lights. It is clear that though these newilluminants do not fall on the locus, they do fall close to the locus.

FIG. 9: SONY DXC-930 normalised camera sensitivities.

FIG. 10: SONY DXC-930 camera data. Log chromaticity differences (LCDs)for 7 surfaces (green, yellow, white, blue, purple, orange and red)under 10 Planckian lights (with increasing temperature from 2800K to10000K). Variation due to illumination is along a single direction.

FIG. 11: SONY DXC-930 camera data. Log chromaticity differences (LCDs)for 7 surfaces (green, yellow, white, blue, purple, orange and red)under 10 Planckian lights (with increasing temperature from 2800K to10000K). LCDs (from FIG. 9) have been rotated so that x co-ordinatedepends only on surface reflectance; the y co-ordinate depends stronglyon illumination.

FIG. 12: Normalised XYZ colour matching curves.

FIG. 13: XYZ tristimuli are transformed to corresponding SONY cameraRGBs using a linear transformation. Log chromaticity differences (LCDs)for 7 surfaces (green, yellow, white, blue, purple, orange and red)under 10 Planckian lights (with increasing temperature from 2800K to10000K) are calculated and rotated according to the SONY rotationmatrix. The x co-ordinate depends weakly on surface reflectance; the yco-ordinate depends strongly on illumination.

FIG. 14: LCDs are calculated for XYZ tristimuli which have beentransformed to corresponding SONY camera RGBs using a linear transformare plotted as filled circles. Lines linking these approximateco-ordinates to the actual SONY LCD co-ordinates are shown.

FIG. 15: 1^(st) and 3^(rd) columns show Luminance grey scale images of aBall and a detergent packet under 5 lights (top to bottom). 2^(nd) and4^(th) columns show corresponding invariant images.

FIG. 16: Equations –5.

FIG. 17: Equations 6–12.

FIG. 18: Equations 13–17.

FIG. 19: Equations 18–22.

In the following description, colour image formation, thelog-chromaticity coordinate space and image variation due to Planckianillumination are discussed. An invariant co-ordinate transform issubsequently derived. For cameras with Dirac delta functionsensitivities and where illumination is modelled by Planck's formuladerivation is analytic. For the practical case of actual cameras andactual illuminations, a statistical technique is also presented.Experimental results are also presented. Various equations referred toin the text are reproduced on the Equation pages forming part of thedrawings.

BACKGROUND

An image taken with a linear device such as a digital colour camera iscomposed of sensor responses that can be described by Equation (1),shown in FIG. 16, in which λ is wavelength, P_(k) is sensor response,k=R, G, B (red, green and blue sensitivity), E is the illumination, S isthe surface reflectance and R_(k) is a camera sensitivity function.Integration is performed over the visible spectrum w.

Let us assume that the camera sensitivity can be expressed as a Diracdelta function, having significant sensitivity only at some wavelengthλ_(k). Then in general R_(k)(λ)=δ(λ−λ_(k)). Dirac delta functions havethe well known sifting property that allow us to rewrite Equation (1) asEquation (2), shown in FIG. 16.

Clearly, under E₁(λ) and E₂(λ) the RGB response for a particular surfacecan be related, thus giving Equation (3), as shown in FIG. 16.

It is to be noted that the diagonal matrix relating the RGBs acrossillumination does not depend on surface reflectance. The same 3 scalarsrelate all corresponding pairs of RGBs. To ease the notation Equation(3) can be rewritten as Equation (4), shown in FIG. 16, where thesubscript denotes dependence on a particular illumination.

One application for the invention is in a process which involveschecking the colour of foodstuffs or other processed materials or thecolour and/or pattern of manufactured articles such as ceramic tiles, atthe end of a manufacturing or processing line.

In judging goods such as food, fruit, or ceramic tiles, it is oftenimportant to replicate typical viewing conditions. The visual appearanceof goods if judged by human experts will often be performed under avariety of different lighting conditions although the human expertinvariably seems to compensate for the different colour temperatures ofthe different illuminations to a certain extend. However the reasoningis that by checking under a variety of different lighting conditions,correlates with the different lighting conditions encountered by theaverage observer (eg average consumer). It follows then that inautomating a quality control/inspection process, it is advantageous toprovide a system that can work in similarly unconstrained viewingconditions. Unfortunately this has proven to be hard to do since thehuman eye and brain carries out some, as yet unknown, intelligentprocessing of images so that some effects produced by different viewingilluminants are discounted. Thus for example, a human being can judgethe ripeness of fruit equally well under daylight conditions (where thelight is bluish) or indoors under incandescent light (where the lightmay be orangish).

The present invention provides a corrected or adjusted colour signal(typically an PGB signal) which is a reflectance correlate which isstable under a plurality of different illuminating light colours (ietemperatures) and intensities. Any image analysis that is based on thiscorrelate performed on the correlated/adjusted signals will, like theprocessing carried out by a human inspector, likewise be unaffected byviewing conditions. An ability to be able to inspect goods underarbitrary lighting conditions will be particularly useful when viewingconditions cannot be held fixed (eg in the checking of harvestedfoodstuffs for colour and/or size and/or shape under non-uniform or evenvarying lighting conditions such as can occur during the day or from dayto day outdoors and/or indoors where outside (natural light) is acomponent of the illumination incident on the foodstuff.

The same varying light conditions can apply when manufactured articlesare being checked for accuracy of pattern, position of pattern, andcolour such as in the case of ceramic tiles.

In both cases, by adjusting and correcting the viewing system outputsignal so as to remove variations due to different coloured illuminationand different intensity of illumination, the resultingcorrected/adjusted signals can be subjected to image analysis in theknowledge that any variation in illumination (colour or intensity) willnot produce incorrect results.

The substitution of a Dirac delta function for R_(k)(λ) has clearlysimplified the image formation equation.

Unfortunately, no camera could possibly, or usefully, be sensitive toonly 3 narrow-band wavelengths of light and Equation (4) does not reallyaccount for image formation in typical cameras. Fortunately, researchhas shown that Equation (4) models image formation fairly well forcameras whose response sensitivities are sufficiently narrow-band. Evenwhen Equation (4) does not hold, it can often be made to hold byapplying an appropriate change or sensor basis. It is pointed out thatthis result is based on a statistical analysis which takes account onlyof “reasonable” illuminating lights and surfaces. This is an importantpoint to bear in mind since, for all but the special case of Dirac deltafunction sensitivities, it is always possible to hypothesise sets ofilluminating lights and surfaces for which Equation (4) will not hold.In practice however, Equation (4) is, or can be made to be, a tolerableapproximation for most real cameras. Henceforth it is assumed thatEquation (4) is a good model of image formation across illumination, andthis has been verified by experiment (see later).

Remarkably, even Equation (4) is an over general model of imageformation. Illumination colour is not arbitrary and so the scalars α, βand γ in Equation (4) are not arbitrary either. Because illuminatinglight is always a positive function (i.e. its power cannot be negative),the scalars themselves must also be positive. However, some positivepower spectra do not occur as illuminations e.g. saturated purpleilluminations do not occur in nature. An implication of this observationis that certain positive triples of scalars are impossible since theyserve only to model the relation between illumination pairs that do notactually occur.

Let us suppose that illumination might be modelled as a black-bodyradiator using Planck's equation (Equation (5)), shown in FIG. 16.

Equation (5) defines the spectral concentration of radiant excitance, inWatts per square meter per wavelength interval as a function ofwavelength λ (in meters) and temperature T (in Kelvin). The constants c₁and c₂ are equal to 3.74183×10⁻¹⁶ and 1.4388×10⁻² mK respectively.

In accompanying FIGS. 1, 2 and 3 the normalised black-body illuminantsfor temperatures of 2500K, 5500K and 10000K are plotted. It is evident(and very well known) that as the temperature increases so the spectrumof light moves from reddish to whitish to bluish. The range 2500K to10000K is the most important region in terms of modelling typicalilluminant colours.

In FIG. 4, the normalised spectral power distribution of a typicalDaylight illuminant (CIE standard daylight source D55) and a 5500KBlack-body radiator are compared. It is clear that the shape of the twoilluminant curves is broadly similar.

In general Planck's equation captures the general shape of incandescentand daylight illuminants. Of course, while the shapes may be similar,Equation (5) does not account for varying illuminant powers. To modelvarying power an intensity constant I is added to Planck's formula,giving rise to Equation (6), shown in FIG. 17.

While the shape of daylight and Planckian radiators is similar, this isnot true for fluorescents (which tend to have highly localised emissionspikes). But, remarkably, even here Equation (6) can be used, becauseinterest lies not in spectra per se, but rather in how they combine withsensor and surface in forming RGB components in the colour signal. Foralmost all daylights and typical man-made lights, includingfluorescents, there exists a black-body radiator, defined in (6), which,when substituted in (1), will induce very similar RGB components formost surface reflectances. Interestingly, if such a substitution cannotbe made, the colour rendering index (broadly, how good surface colourslook under a particular light) is poor. Indeed, the lighting industrystrives to manufacture lights such that their chromaticities lie closeto the Planckian locus and which induce RGB components, for most surfacereflectances, which are similar to those induced by a correspondingblack-body radiator.

In FIG. 5 is plotted the xy chromaticity diagram. The solid curving linemaps out the chromaticity locus of Black-body radiators from 100K to20000K (from right to left). The chromaticities have also been plottedfor 36 typical illuminants (including daylights and artificial lights)measured around the Simon Fraser University campus. It is evident thatall of these illuminants fall very close to the Planckian locus.

Perhaps more interesting is the question of whether a Planckianillumination substituted for the fluorescent illumination in Equation(1) would render similar colours. To evaluate this the followingexperiment was carried out.

Using the XYZ standard observer functions for the human visual systemCIE Lab coordinates were calculated for the 170 object reflectancesmeasured by Vrhel et al (Colour Reasearch and Application 19: 4–9, 1994)under each of the 36 SFU illuminants (Lab coordinates are a non-linearfunction of XYZ tristimuli). The CIE Lab coordinates were thencalculated for the same surfaces when a Planckian illuminant (a spectracorresponding to Equation (6)) is substituted for the actual illuminant.The Euclidean distance, ΔE error, between the Lab coordinates for thetrue illuminant, and the Planckian substitute, were calculated for eachsurface. (Euclidean distances in CIE Lab roughly correlate withperceived perceptual differences). For each light the mean ΔE wascalculated between the actual and Planckian substituted Lab coordinates.The average over all 36 means was found to be 2.24 with a maximum meanof 6.44.

Meyer et al have found that mean ΔEs of as large as 5 or 6 areacceptable in image reproduction i.e. images are similarly andacceptably rendered. One may thus reasonably conclude, that for 36illuminants, a Planckian substitution can be made (the perceivedperceptual rendering error is quite small).

Of course, the XYZ human observer sensitivities are not in general thesame as digital camera sensitivities. That is, accurate rendering forthe visual system need not imply accurate rendering for a camera.Fortunately, the responses of most cameras (and certainly all thoseknown to the Inventors) are, to a tolerable approximation, transformableto corresponding XYZs. Accurate rendering of XYZs, for reasonablesurface reflectances, implies accurate rendering of RGBs.

Colour Constancy at a Pixel

Considering next the above model of image formation together withPlanck's equation it can be shown that there exists one coordinate ofcolour, a function of RGB components, that is independent of lightintensity and light colour (where colour is defined by temperature).However, to make the derivation cleaner a small (and often made)simplifying alteration to Equation (6) is made. In Planck's equation λis measured in meters; thus we can write wavelength λ=x*10⁻⁷ where x ∈[1, 10] (the visible spectrum is between 400 and 700 nanometer 10⁻⁹).Temperature is measured in thousands of degrees Kelvin or equivalentt*10³ (where t ∈ [1, 10]). Substituting into the exponent of Equation(6) gives Equation (7), shown in FIG. 17.

Because t is no larger than 10 (1000K) and there is no significantvisual sensitivity (for humans or most cameras) after 700 nm, x≦7, itfollows that

${e^{\hat{}}\frac{1.4388*10^{2}}{x\; t}}\operatorname{>>}1$and Equation (8) follows.

FIGS. 1 to 3 contrast the Black-body illuminants derived using Equation(8), shown in FIG. 17, with those defined by Equation (6) fortemperatures of 2500K, 5500K and 10000K. It is clear that theapproximation is very good (in particular the two spectra for 2500Koverlay each other exactly).

Substituting Equation (8) in Equation (2) gives Equation (9), shown inFIG. 17.

Taking natural logarithms of both sides of Equation (9), shown in FIG.17, gives Equation (10), shown in FIG. 17.

That is, log-sensor response is an additive sum of three parts: ln I(depends on the power of the illuminant but is independent of surfaceand light colour); ln(S(λ_(k))λ⁻⁵ _(k) c₁) which depends on surfacereflectance but not illumination) and

$- \frac{c^{2}}{T\;\lambda}$(which depends on illumination colour but not reflectance).

Remembering that, in Equation (10), P_(k)=R,G,B; there are threerelationships which exhibit the same structure: each of the lnR, lnG andlnB sensor responses are an additive sum of intensity, surface andillumination components. By cancelling common terms, it can be shownthat two new relations can be derived which are intensity independent(but depend on illumination colour) and from these a final relationwhich depends only on reflectance.

Begin by introducing the following simplifying notation: letS_(k)=ln(S(λ_(k))λ⁻⁵ _(k) c₁) and

$E_{k} = {- \frac{c^{2}}{T\;\lambda}}$(k=R, G, B sensor). The following two relations, red and green, and blueand green log-chromaticity differences (or LCDs), are independent oflight intensity:

Note that Equation (11), shown in FIG. 17, is basically a recapitulationof a result that is well known in the literature. There are many ways inwhich intensity or magnitude might be removed from a 3-d RGB vector. Theone that makes the subsequent colour temperature analysis easier isselected.

It is useful to think of Equation (11) in terms of vectors ie:

p_(R)^(′) p_(B)^(′)is a sum of the vector

S_(R) − S_(G) S_(B) − S_(G)plus the vector

E_(R) − E_(G) ${\frac{1}{T}{{E_{B} - E_{G}}}};$where

$\frac{1}{T}$is a scalar multiplier.

Written in this form Equation (11) is the equation of a line written invector notation. In changing surface reflectance, only the first term onthe right-hand side of Equation (11) changes. That is, the lines definedby different surfaces should all be simple translations apart.

Of course, this result is predicated on the approximate Planckian modelof illumination Equation (8). To test if this result holds for realPlanckian black-body illuminants (Equation (6)), sensor responses forseven surfaces under 10 Planckian lights (using narrow-band sensorsanchored at 450 nm, 540 nm and 610 nm) are numerically calculated, usingEquation (1). The 7 surfaces comprise the Macbeth colour checkerreflectances labelled green, yellow, white, blue, purple, orange andred. The 10 Planckian illuminants were uniformly spaced in temperaturefrom 2800K to 10000K. The LCDs (Equation (11)) were calculated and theresulting 2-dimensional coordinates are plotted in FIG. 6. As predicted,as the illumination changes the coordinates for a given surface span aline and all the lines are related by a simple translation.

Now using the usual rules of substitution it is also a simple matter toderive a relationship that is independent of temperature. Thisrelationship is shown in Equation (12), shown in FIG. 17, where allS_(k) and E_(k) are independent of illuminant colour and intensity.Equation (12) shows that there exists a weighted combination of LCDsthat is independent of light intensity and light colour. It is therebyshown that the 1-dimensional colour constancy problem at a pixel can besolved.

It is useful to visualise the geometric meaning of Equation (12). For aparticular surface, all LCDs for different lights fall on a line y=mx+cor in parameterised coordinate form (x, mx+c). In Equation (12) a linearcombination of the x and y coordinates is calculated: (a′x+b′(mx+c)),

$\left( {{{where}\mspace{14mu} a^{\prime}\mspace{14mu}{and}\mspace{14mu} b^{\prime}\mspace{14mu}{are}\mspace{14mu}{the}\mspace{14mu}{constants}\mspace{14mu} 1\mspace{14mu}{and}}\mspace{14mu} - \frac{E_{R} - E_{G}}{E_{B} - E_{G}}} \right).$Clearly if we scale a′ and b′ by some term v giving va′ and vb′, theilluminant invariance of (12) is unaltered. Without loss of generalityv, a=va′, b=vb′ can be chosen, such that the vector [a b]^(t) has unitlength.

Calculating the illuminant invariant as the vector dot product (‘.’)gives Equation (13), shown in FIG. 18.

The meaning of Equation (13) is geometrically well understood: thelog-difference coordinate is projected onto the axis [a b]; where thisaxis is chosen to be orthogonal to the direction of the variation due toillumination.

The following Equation (14), shown in FIG. 18, rotates thelog-difference coordinate axis so that the resulting x axis recordsilluminant independent information and the y axis captures all thevariation due to illumination.

The log difference data shown in FIG. 6 is rotated according to Equation(14). The result is shown in FIG. 7.

Approximate Invariance

If a camera has Dirac Delta functions then the invariant co-ordinatetransform can be calculated analytically. When camera sensorsensitivities are not perfect Dirac Delta functions (and they rarelyare) then the best illuminant invariant quantity must be foundstatistically.

There are two steps involved in finding an invariant. First it isnecessary to make sure that the camera response across illuminationfollows the diagonal matrix model of Equation (4). Second, an equationof the form of Equation (14) must be found.

Worthey and Brill (Heuristic analysis of von Kries colourconstancy—Journal of the Optical Society of America A,3: 1708–1712,1986) found that so long as a camera is equipped with fairly narrowsensitivities, e.g. with a support of 100 nm, the diagonal model willhold. When sensitivities are significantly broader e.g. in excess of 300nm in the case of the human visual system the simple model of Equation(4) does not hold. However, in a series of works, Pinlayson, Drew andothers (see Spectral Sharpening: Sensor transformations for improvedcolour constancy: Journal of the Optical Society of America a,11(5):1553–1563, May 1994) found that new narrower-band sensitivities could beformed from broad band sensitivities by calculating an appropriatesharpening transform. The diagonal model, relative to the sharpenedsensors, once again is quite accurate. Effective sharp transforms havebeen shown to exist for the broad band sensitivities of the human conesand the spectrally broad band Kodak DCS 470 camera (and all other broadband sensor sets known to the Applicants). Henceforth Equation (4) isassumed applicable.

It is pointed out that Equation (4) is a necessary, not sufficient,condition for the analysis to proceed. In the strictest sense thediagonal model of illumination change must occur in tandem with devicesensitivities integrating spectral stimuli like Dirac Delta functions.This was in fact found to be true for a sharp transform of XYZsensitivities. However, the statistical method set forth below can nowbe applied without explicitly enforcing the Delta function equivalence.

To discover the appropriate invariant, it is necessary to understand howLCDs for different surfaces, under Equation (4), relate to one anotherand more specifically how this relationship can be used as a means forfinding the best invariant. It is possible to consider an image of aparticular surface reflectance S^(J)(λ) under a set of representativeilluminants: E₁(λ), E₂(λ) . . . E_(m)(λ). Using Equations (1) and (11),the set of m LCD vectors Q₁ ^(i), Q₂ ^(i), . . . Q_(m) ^(i) can becalculated. Assuming that the camera behaves approximately like a DiracDelta camera, then the LCDs should all be approximately co-linear. Bysubtracting the mean LCD, this line can be moved so that it passesthrough the origin, giving Equations (15a) and (15b), each shown in FIG.18.

Because of the invariance properties derived and discussed hitherto, themean-subtracted points for another surface S^(J)(λ), q₁ ^(j), q₂ ^(j), .. . q_(m) ^(j) must also lie on a similarly orientated line (which againpasses through the origin). Now taking n representative surfaces, n setsof mean-subtracted LCDs can be generated and placed in the columns of a2×nm matrix M, Equation 16, shown in FIG. 18.

The co-variance matrix of this point set is given by Equation (17),shown in FIG. 18.

Equation 14 cast the invariant computation as a rotation problem in LCDspace. We do likewise here and note that if we can find a rotationmatrix R such that the rotated points

${R{\underset{\_}{q}}_{k}^{i}} = \begin{bmatrix} \in \\x\end{bmatrix}$(where ∈ is as small as possible) then good invariance should follow,since Equation (18) applies, where x denotes an illuminant varyingquantity which is not of interest. Under the assumption that ∈ is small,the first coordinate of the rotated LCDs will be approximately invariantto illuminant change: it is equal to the dot-product of the first row ofR with the mean vector μ^(i).

To find the rotation satisfying Equation (18), shown in FIG. 19, it isnecessary to find the co-ordinate axis along which the variation (orvariance) due to illumination is minimum. To do this first note thatcovariance matrix Σ(M) can be uniquely decomposed, as in Equation (19),shown in FIG. 19, where U is a rotation matrix and D is a strictlypositive diagonal matrix where the diagonal terms of D₁, D_(1,1)=σ₁ ²and D_(2,2)=σ₂ ², are ordered: σ₁ ²>σ₂ ². Simple algebra establishesEquation (20), shown in FIG. 19.

Thus, the diagonal entries of D are the variances of M under rotation U.Furthermore, over all choices of rotation matrix U, it can be shown thatσ₁ ² is the maximum variance that can be achieved and σ₂ ² the minimum.It follows then that R can be defined in terms of U: the first andsecond rows of R equal the second and first rows of U ([R₁₁ R₁₂]=[U₂₁U₂₂] and [R₂₁ R₂₂]=[U₁₁ U₁₂]).

Invariance Across Devices

In setting forth the above (both the analytic and statistical methods),it is clear that the derivation is camera specific. That is, theinvariant information that is available changes with the imaging device.It is interesting however, to ask whether the similar invariantinformation might be calculable across different cameras. The answer tothis question is trivially true if there exists a one to one mappingthat takes the RGB response of an arbitrary camera to a correspondingRGB response for some fixed canonical camera. Unfortunately, such amapping, save under very restrictive circumstances, cannot exist for alllights and surfaces; there will always exist pairs of physical stimulithat induce identical responses (they are meters) with respect to onecamera but induce quite different responses with respect to a secondcamera. However, in practice, the one to one mapping holds for mostreasonable stimuli.

Drew and Funt (Natural Metamers: CVGIP:Image Understanding, 56:139–151,1992) demonstrated that if the colour signal (the product of light andsurface reflectance) spectra are described by a 3-dimensional linearmodel (approximately true for many natural reflectance spectra viewedunder daylight) then RGBs across two different cameras are linearlyrelated. This work has been generalised with the understanding that itis not spectra per se which are of interest but how they project downonto RGBs. Empirical evidence has been presented which shows that therecorded RGBs, for a large corpus of physical stimuli and a variety ofcolour devices, are linearly related (to a good approximation).

It follows that to find an invariant co-ordinate across devices it isappropriate to:

-   1. Find the linear transform T which best maps the RGBs for a given    device to corresponding RGBs for a canonical device:    Tp ^(d) ≈p ^(c)    where the superscripts d and c denote a given device and the    standard reference canonical device.

2. Calculate for the canonical device the best invariant can becalculated using either of the methods set forth in Equations (7) to(20). Denoting the invariant calculation as, the function f: R×R×R→R ,then invariant information for p ^(d) is calculated as:f(Tp ^(d))≈f( p ^(c))

If the canonical device is chosen to be a hypothetical camera that isequipped with Dirac Delta function sensitivities then T can be thoughtof as a sharpening transform.

Multiple Illuminants

Often in colour constancy research it is interesting to considersituations where there are multiple illuminants present in a scene. Solong as the effective illuminant (which may be a combination of themultiple light sources) lies on the Planckian locus, the derivation ofEquations (7) to (11) applies and invariance is assured. However, it isreasonable to assume that there may be some averaging of light sources.For example, if E₁(λ) and E₂(λ) are spectrally different light sourcesboth incident at a point in a scene, then due to the superposition oflight, the effective illumination E(λ) is equal to E₁(λ)+E₂(λ). Assumingthat E₁(λ) and E₂(λ) lie on the Planckian locus and have temperatures T₁and T₂ and intensity constants I₁ and I₂, then Equation (21), shown inFIG. 19, applies.

Because the Planckian locus is convex, no Planckian illuminant can bewritten as a convex sum of any other two Planckian. For all choices ofconstants I₃ and T₃, Equation (22), shown in FIG. 19, results.

Fortunately, the Planckian locus, in the region that spans typicalilluminants is only very weakly convex. As such, while additivecombinations of Planckian illuminants cannot lie on the Planckian locus,they will lie close to the Planckian locus. In FIG. 8, we have plottedthe xy chromaticity diagram, the Planckian locus between 2800K and10000K (the range of typical lights) and the chromaticities of 190illuminants (created by taking weighted convex combinations of Planckianilluminants in this range). All lights lie on or close to the Planckianlocus. That is, we expect the derived illuminant invariant to be stableeven when there are additive light combinations.

Experiments

FIG. 9 shows normalised spectral sensitivity functions of a SONY DXC-9303-CCD digital camera. It is evident that these sensors are far fromidealised delta functions. Each is sensitive to a wavelength intervalover 100 nm in extent. Thus, at the outset it is necessary to test theadequacy of the illumination invariant representation. Using Equation(1) the SONY camera response for seven surfaces under 11 Planckianlights were generated. As in FIG. 6 (the experiment using Dirac Deltafunction sensitivities), green, yellow, white, blue, purple, orange andred reflectances drawn from the Macbeth colour checker were used alongwith 10 Planckian black-body illuminants evenly spaced in temperaturefrom 2800K to 10000K. The corresponding log-chromaticity differences(LCDs) were calculated and the resulting 2-dimensional coordinates areplotted in FIG. 10. The rotated co-ordinates, calculated using theapproximate invariant method outlined hitherto, are shown in FIG. 11.

It is evident that for camera responses the variation inlog-chromaticity due to illumination is not spread along a singledirection. The calculated invariant (Equation (13)) is only approximate.Yet, for particular surfaces the LCDs do fall on a line. Moreover, whilethe direction of these lines do vary as a function of surface colourthey do not vary much. We conclude then that for the SONY camera it ispossible to calculate a single colour feature that has very weakdependence on illumination.

To quantify the magnitude of the dependence the following experiment wascarried out.

Using the technique hitherto set forth under Approximate Invariance, theoptimal invariant for 170 object reflectances and the 10 Planckianilluminants are calculated. For the ith reflectance calculate σ_(i) ²;namely the variance of the invariant co-ordinate over the 10 lights. Thesum of all 170 individual variances gives us a measure of the errorvariation in the signal: σ_(E) ²=Σ_(i)σ_(i) ². The total variancecalculated for the invariants of all 170 surfaces viewed under the 10lights (the signal variance), is denoted: σ_(S) ². The signal to noiseration, σ_(S)/σ_(E), advises how large the signal is relative to theerror. For this data the SNR was found to be approximately 23, that isthe signal is 23 times as large as the noise. Two informal conclusionscan be drawn. First, that the invariant is sufficiently stable to allow,on average, up to 23 colours to be distinguished from one another.Second, that the invariant conveys slightly more than 4 ‘bits’ ofinformation.

By definition the invariant is chosen that minimises σ_(E). However, itwould be equally possible to find the co-ordinate direction thatmaximises the variation due to illumination e.g. the vector [U₁₁ U₁₂} inEquation (19) which is in the direction orthogonal to the calculatedinvariant. Relative to this worst case co-ordinate, the calculatedinvariant can be expected to have a much lower signal to noise ratio.Indeed, the ratio was found to be 1.9. That is, the signal variation dueto reflectance is only twice as large as that due to illumination.Informally, it would only be possible to reliably discriminate twocolours.

This experiment was repeated using the 10 Planckian lights and 180 newilluminants formed by taking additive combinations of Planckianilluminants (see FIG. 8). These new lights simulate common lightingconditions such as outside daylight mixing with indoor incandescentillumination. As discussed in Invariance across devices, most of theselights necessarily lie off-locus and so the invariant calculation is inthis case only approximate and so a lower SNR can be expected. Thesignal to noise ratio was found to be reduced from 23 to 15. Thisindicates that the calculated invariant is quite robust to many typicallights and lighting conditions (it is still possible to calculate 4 bitsof information).

Being interested in evaluating the similarity of invariant informationcalculated across devices, using the XYZ colour matching functions,plotted in FIG. 12, the XYZ response was calculated for the red, green,yellow, purple, white, blue and orange patches under the 10 Planckianilluminants. The best 3×3 matrix transform mapping XYZs to correspondingSONY RGBs was found. Finally the LCDs were calculated and rotatedaccording to the rotation matrix derived for the SONY camera. Theresulting co-ordinates are shown in FIG. 13 and are contrasted with theactual SONY co-ordinates in FIG. 14. It is evident that deviceindependent information is calculable. Though, some reflectances areclearly less stable than others.

In order to test the invariant calculation on real camera images, 10SONY DXC-930 images (from the Simon Fraser dataset) of two colourfullobjects (a beach ball and a detergent package) were taken under thefollowing 5 illuminants:

-   Macbeth fluorescent colour temperature 5000K (with and without blue    filter),-   Sylvania Cool white fluorescent,-   Philips Ultralume,-   Sylvania Halogen.

These illuminations constitute typical everyday lighting conditions;i.e. yellowish to whitish to bluish light. The luminance grey-scaleimages, calculated by summing R+G+B, are shown in the first and thirdcolumns of FIG. 15. It is clear that the simple luminance grey-scale isnot stable across illumination. In columns 2 and 4 the correspondinginvariant grey-scale images are calculated. It is equally clear that thegrey-scale pixels in these images do not change significantly as theillumination changes. Also notice that qualitatively the invariantimages maintain good contrast. Not only has illumination invariance beenobtained but the images that result are visually salient.

As a concrete test of the utility of the calculated illuminant invarianta set of object recognition experiments were carried out. To the beachball and detergent packages were added 9 other colourful objects. Thesetoo were imaged under all 5 lights. For all 55 images their respectivegrey-scale invariant histograms were calculated and then used as anindex for object recognition. Specifically each light was taken in turnand the corresponding 11 object histograms used as feature vectors forthe object database. The remaining 44 object histograms were matchedagainst the database; the closest database histogram being used toidentify the object.

It was found that a 16 bin invariant grey-scale histogram, matched usingthe Euclidean distance metric, delivers near perfect recognition.

Almost 96% of all objects were correctly identified. Moreover, thoseincorrectly matched, were all found to be the second best matchingimages.

This performance is really quite remarkable. Funt et al, hithertoreferred to, measured the illumrinant using a spectra-radiometer andthen corrected the image colours based on this measurement (so calledperfect colour constancy). Objects were indexed by matching correctedchromaticity histograms. Surprisingly, only 92.3% recognition could beachieved. Moreover, at least one object was matched in 4^(th) place (thecorrect matching histogram was the fourth best answer).

That 1-d invariant histograms apparently work better than 2-d colourhistograms is at first glance hard to understand. Two explanations havebeen considered. Given a measurement of the light it is not possible toexactly map RGBs from one lighting condition to another and so perfectmatch performance cannot be expected. However, the performance that isseen must depend on the properties of the colour space being used. Thatis, a first possible explanation is that if a 1-d coordinate, such asthe invariant presented here, is relatively robust (relative to otherchoices of colour space) to illuminant change (there was negligiblemapping error for that coordinate) then it might be expected that theinvariant coordinate would support relatively good performance.Unfortunately, this explanation though appealing (it would be pleasingif our derived invariant was found to have additional nice properties)was not found to be true. Indeed, it was found that the absoluteresidual error found after correcting for the illuminant was actuallyrelatively high in the direction of the derived invariant. That is,viewed from a mapping error perspective alone, one might expect the 1-dhistograms to support poorer indexing.

The second possible explanation rests on the colour space being used. Inthe original experiments of Funt et al, previously referred to, thetraditional rg chromaticity space, which is bases on R, G and B cameraresponses, is used. In this invention the derived invariant is base onIn R, In G and in B responses. Experiments, reported elsewhere, on thesame data set, have found that a log chromaticity space supports moreaccurate indexing than conventional chromaticity space. It is speculatedthat indexing based in log colour space might work better because thelogarithm function maps raw RGBs to more perceptually meaningfulquantities; since Euclidean colour differences in log space are bettercorrelated to pereceived colour differences, (the logarithm functionenforces Weber's law). Ultimately the objects discriminated in theexperiments herein were coloured in such a way that look distinct anddifferent to the eye, so matching should be perceptually relevant.

Funt et al, hitherto referred to, also used a variety of colourconstancy algorithms, including max RGB, grey-world and a neural netmethod, as a pre-processing step in colour distribution basedrecognition. All methods tested performed significantly worse than theperfect colour constancy case. No algorithm delivered supported morethan a 70% recognition rate.

Other colour invariant based methods, predicated on functions of manyimage pixels, have also been tried on the same data set. None deliveredresults better than the 96% recognition rate achieved with the presentinvention.

CONCLUSIONS

Colour constancy, in its most general form amounts to re-rendering agiven RGB image so that it appears as if it were obtained using a knownreference light. This problem has turned out to be very hard to solve somost modern algorithms attempt only to recover reference lightchromaticities. This simpler 2-dimensional colour constancy problemthough more tractable is only soluble given sufficient colour content inthe scene. In this specification, it has been considered if a1-dimensional colour constancy problem might be easier to solve.Specifically, there was sought a single image colour co-ordinate, afunction of RGB, that might be easily mapped to the known referenceconditions.

It has been shown that such a coordinate exists—namely a particularlinear combination of log RGB responses, which has been shown to beinvariant to light intensity and illuminating light colour. Moreover, byconstruction, the invariant co-ordinate under reference and all otherlights remains unchanged so no colour constancy mapping is actuallyneeded (the mapping is always the identity transform). The result restson two assumptions: that camera sensitivities behave like Deltafunctions and illumination chromaticities fall near the Planckian locus.Many cameras have sensitivities which are sufficiently narrow that theybehave as if they were equipped with Delta function sensitivities. Whena camera has broad-band sensitivities, then a basis transform usuallysuffices to take camera measurements to a co-ordinate system where theDelta function assumption holds. The Inventors have yet to find a camerawhere the invariant calculation cannot be made.

Experiments show that invariant co-ordinate (coded as a grey-scale)histograms provide a stable cue for object recognition. Indeed, theysupport better indexing performance than chromaticity histograms createdpost-colour constancy processing.

1. A method of processing an N-component color signal from a colorsignal source so as to eliminate dependency on color temperature and/orintensity variation by using a mathematical algorithm derived from thesource sensitivities to different wavelengths of light in a signalprocessing computer supplied with the color signal, to provide newvalues for the components making-up the color signal which areindependent of the color temperature of the light illuminating, ordeemed to illuminate, the subject described by the color signal, whereinthe N components, such as RGB signals of a 3 component color signal,define an image of an illuminated colored subject, the processingserving to produce modified signal components defining an image of thesame subject as if illuminated under light of different colortemperature, comprising the steps of: (1) generating the N-(RGB etc.)components of a color signal corresponding to an illumination of thesubject using illumination of color temperature T₁; (2) forming the Nlog values (R′ G′ B′ etc.) of the N-RGB etc. components of the colorsignal so obtained; and (3) computing the N log values (R*, G*, B* etc.)of the N-RGB etc. components of the color signal for a differentillumination color temperature T₂ using the equation:(R*G*B* . . . )=(R′G′B′. . . )+X(u,v,w, . . . ) where X is proportionalto (T1–T2) and u, v, w, etc. are constants that depend only on thespectral sensitivities of the signal source and Planck's equationdefining black body illumination.
 2. A method as claimed in claim 1further comprising the step of converting the log values (R*, G*, B*, .. . ) to N different signals, typically appropriate RGB etc. values, forprocessing and/or display and/or storing.
 3. A method as claimed inclaim 1 wherein the color signal is derived from a camera, multispectralimaging device, or a scanner, or is a computer generated signal.
 4. Amethod as claimed in claim 1 in which the method is applied to the N-RGBetc. values of each pixel making up a multi-pixel image.
 5. A correctedRGB signal when obtained by the method of claim 1, comprising threeparts namely in which the two non-zero log values are independent of theillumination color temperature.
 6. A corrected RGB signal when obtainedby the method of claim 1, in the form of a vector (R*, G*, B*)−G*/v(u,v, w) which can be written in the form (χ,

, γ) in which the non-zero components χ and γ are two values that areindependent of the color temperature of the illuminant.
 7. A method asclaimed in claim 1, wherein the N values are normalized with respect toone of the N values to remove intensity dependence and reduce the numberof values to (N−1).
 8. A method as claimed in claim 7 wherein there arethree components R G and B in the original signal and as a result ofnormalisation these are reduced in number to R/G and B/G, and colorintensity independence is obtained by converting these two components,one to zero and the other to (B′−R′ b/a) where B′ is log B/G and R′ islog R/G, and the values of a and b are derived from the values u, v, wfor the signal source.
 9. A method as claimed in claim 1, comprising thestep of calibrating the source in advance to determine the (u, v, w etc.(or a, b, c etc.)) values for the source.
 10. A computer when programmedso as to perform the method of claim
 1. 11. A method as claimed in claim1, further comprising the step of converting corrected component signalvalues as aforesaid into a grey scale or pseudo color signal forprocessing and/or comparison and/or storage and/or for forming a greyscale or pseudo color image in place of a true-color image in a display.12. A method as claimed in claim 11 wherein a 1-dimensional colorco-ordinate is expressed as a function of chromaticity, so as to removedependence on the illumination color temperature by converting an 2component color signal to a single component grey-scale signal or pseudocolor signal, and a comparison is performed by the step of mapping thegrey-scale values (or pseudo color values) to grey-scale values (orpseudo color values of a similar scene observed under reference lightingconditions.
 13. A method of color-based object recognition employingcolor signals as processed and/or normalised as claimed in claim 11wherein objects are represented by the distributions of invariantco-ordinates calculated from color images of the objects obtained by theprocessing to produce grey-scale histograms (since the invariantco-ordinates are coded as grey-scale), and recognition is performed bydistribution comparison whereby the distribution of an unknown object iscompared with known object distributions stored in a database, and theclosest match identifies the unknown object.
 14. A method of imageanalysis comprising the steps of comparing illumination colortemperature and/or intensity corrected signals obtained by the method ofclaim 1 or summations or other computational combinations of some or allof the intensity corrected signals, with stored values of such signalsand/or similar summations or other computational combinations thereof,to permit object recognition.
 15. A computer programmed to perform thecomparison set forth in claim
 14. 16. A method as claimed in claim 1 inwhich the source includes RGB etc. correction in each of the N channelsfor the N components to compensate for the expected non-linearity of adisplay device such as CRT based display or monitor to which the signalcould be supplied, and a linear relationship between the N-componentcolor signal and the resulting image is obtained by dividing the logcamera responses by the average of the log color responses in a localneighbourhood of the pixel of interest in the image prior to calculatingthe invariant co-ordinates i.e. corrected component values.
 17. A methodas claimed in claim 1 in which after the invariant co-ordinate(s) is/arecalculated, the value(s) is/are normalised by the average invariantco-ordinate in a local neighbourhood of a pixel of interest in theimage.
 18. A method of processing an N-component color signal (N>1) froma color signal source so as to eliminate dependency on color temperatureand/or intensity variation by using a mathematical algorithm derivedfrom the source sensitivities to different wavelengths of light in asignal processing computer supplied with the color signal, to providenew values for the components making up the color signal which areindependent of the color temperature of the light illuminating, ordeemed to illuminate, the subject described by the color signal, themethod comprising the step of adjusting the N components of the signalso as to produce component values which when using an imaging means toimage a colored subject are independent of the color temperature of thelight illuminating the subject wherein the adjustment is effected by thestep of computing for each of (N−1) components a new value using theoriginal component value, the Nth component value, and an algorithmincorporating a mathematical operator whose value is related to thespectral sensitivity of the imaging means to different color temperatureilluminants.
 19. A method of processing an N-component color signal(N>1) as claimed in claim 18, wherein the N values are normalized withrespect to one of the N values to remove intensity dependence and reducethe number of values to (N−1).
 20. A method as claimed in claim 19,wherein there are three components R G and B in the original signal andas a result of normalization these are reduced in number to R/G and B/G,and color intensity independence is obtained by converting these twocomponents, one to zero and the other to (B′−R′.b/a) where B′ is log B/Gand R′ is log R/G, and the values of a and b are derived from the valuesu, v, w for the signal source.
 21. A method of image enhancement whenprocessing a color signal in a system in which the log of the colorresponse is composed of three parts, a vector due to the intensity ofthe illumination, a vector due to the illumination color temperature,and a vector due to reflective properties of the surface illuminated forthe purpose of producing the image, comprising the step of adjusting themagnitude of the intensity vector, or the magnitude of the illuminationcolor temperature vector for each pixel so as to change the effectiveillumination for each pixel in a display of the image using theprocessed signal.
 22. A method of enhancement as claimed in claim 21further comprising the step of increasing or decreasing the colorwarmness or color depth by altering the magnitude of the illuminationcolor temperature vector of the signal.
 23. A method as claimed in claim22 further comprising the step of altering the magnitude by scaling allillumination color temperature vectors for all pixels in the image. 24.A method as claimed in claim 22 further comprising the step ofincreasing or decreasing the magnitude of the illumination intensityvector for all pixels in an image so as to brighten or darken theoverall picture, to allow enhancement of a final image without alteringthe original illumination.
 25. A method as claimed in claim 22 whereinsaid step of adjusting the magnitude of the intensity vector isperformed on a frame basis, or on selected regions of a frame made up ofmany pixels, or on a pixel by pixel basis.
 26. A method of imageenhancement applied to N-component color signal, such as the RGB signalsof a 3 component color signal, defining an image of an illuminatedcolored subject, wherein the components are processed to producemodified signal components defining an image of the same subject as ifilluminated by a different color temperature (T), comprising the stepsof: (1) generating the N-(RGB etc.) components of a color signalcorresponding to an illumination of the subject using illumination ofcolor temperature T; (2) normalising the N components of the colorsignal with respect to one of the components (e.g. G) to provide (N−1)co-ordinates, such as R/G, B/G in a 3 component RGB signal, so that eachof the resulting components is independent of illumination intensity,(3) computing the log values of these (N−1) co-ordinates, such thatlog(R/G)=R′ and log (B/G)=B′, in the case of an RGB signal, and (4)adjusting the values of R′, B′, etc., to produce new values R*, B*,etc., corresponding to the values which would apply if the colortemperature of the illumination is T₂ using the equation (R*, B*,etc.)=(R′, B′, etc.)+X(a, b, etc.) where X is proportional to (T₁–T₂)and a and b etc. are constants that depend only on the spectralsensitivities of the source and Planack's equation defining black-bodyillumination.
 27. A method as claimed in claim 26 further comprising thestep of converting the values R*, B*, etc., to new RGB etc. values, andusing the new values for processing and/or comparison and/or storageand/or display.
 28. A method as claimed in claim 26, wherein said signalhas a source and wherein said source is a camera, a multispectralimaging device, a scanner or a computer programmed to produce a colorsignal.